User interface for displaying internal state of autonomous driving system

ABSTRACT

Autonomous vehicles use various computing systems to transport passengers from one location to another. A control computer sends messages to the various systems of the vehicle in order to maneuver the vehicle safely to the destination. The control computer may display information on an electronic display in order to allow the passenger to understand what actions the vehicle may be taking in the immediate future. Various icons and images may be used to provide this information to the passenger.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 14/333,804, filed Jul. 17, 2014, which is a continuation ofU.S. patent application Ser. No. 13/781,113 filed Feb. 28, 2013 now U.S.Pat. No. 8,818,610, which is a continuation of U.S. patent applicationSer. No. 13/682,920 filed Nov. 21, 2012 now U.S. Pat. No. 8,706,342,which is a continuation of U.S. patent application Ser. No. 12/769,252filed Apr. 28, 2010 now U.S. Pat. No. 8,346,426, the entire disclosuresof which are hereby incorporated herein by reference.

BACKGROUND OF THE INVENTION

Field of the Invention

The invention relates generally to user interface applications forautonomous driving systems. More specifically, user interfaces fordisplaying the status of the autonomous driving system are provided.

Description of Related Art

Autonomous vehicles use various computing systems to transportpassengers from one location to another. Some autonomous vehicles mayrequire some initial input from an operator, such as a pilot, driver, orpassenger while other systems may require continuous input. Othersystems, for example autopilot systems, may be used only when the systemhas been engaged, thus the operator may switch from a manual to anautonomous mode where the vehicle drives itself.

A key component of an autonomous vehicle is the perception system, whichallows the vehicle to perceive and interpret its surroundings during atrip. When the autonomous system is engaged, the system will makevarious decisions during the trip, for example, speed up, slow down,stop, etc. The operator may be unaware of the calculations or“reasoning” behind why the autonomous vehicle is taking some particularaction. In order to feel safe and confident, the operator may want toknow what the vehicle is planning to do in the immediate future.

Navigation systems may include electronic displays which appear to zoomin or out according to a vehicle's speed of travel to enable to user toidentify where the vehicle may be within the next few seconds. Some ofthese systems provide real-time traffic information received via radioor satellite signals. However, these systems do no provide for thedisplay of the speed or actual location of other vehicles or obstacles.

BRIEF SUMMARY OF THE INVENTION

An aspect of the invention provides a vehicle. The vehicle includes aplurality of control apparatuses including a braking apparatus, anacceleration apparatus, and a steering apparatus; a user input devicefor inputting destination information; a geographic position componentfor determining the current location of the vehicle; an object detectionapparatus for detecting and identifying a type of an object in orproximate to a roadway; memory for storing a detailed roadway mapincluding roadways, traffic signals, and intersections; an electronicdisplay for displaying information to a passenger; and a processor. Theprocessor is programmed to receive the destination information; identifya route to the destination; determine, from location informationreceived from the geographic position component and the stored mapinformation, the current geographic location of the vehicle; identify anobject and object type based on object information received from theobject detection apparatus; determine an action to be taken includingcontrolling at least one of the control apparatuses based on theidentified object, the current geographic location of the vehicle, andthe route; and select images to be displayed based on the action to betaken and the identified object.

In one example, the identified object is a second vehicle and the imageto be displayed is an icon representing the second vehicle.

In one example, the processor is further programmed to display theselected objects on the electronic display.

In one example, the icon representing the second vehicle is selectedbased on the type of vehicle.

In one example, the processor is further programmed to display, on thedisplay, an image indicting a portion of the route to be traveled by thevehicle in the next few seconds.

In one example, the identified object is a second vehicle and the actionto be taken is maintaining a safe following distance behind the vehicle.

In one example, the identified object is a second vehicle and the actionto be taken is avoiding a headroom zone in front of the second vehicle.

In one example, the action to be taken is waiting and the method furthercomprises displaying text indicating that the vehicle is waiting.

In one example, the processor is further programmed to determine ageographic area to be displayed such that a larger geographic area isdisplayed where the vehicle is moving faster, and a smaller geographicarea is displayed where the vehicle is moving slower.

In one example, the processor is further programmed to determine ageographic area to be displayed such that a larger geographic area isdisplayed where the roadway is associated with a relatively high speedlimit, and a smaller geographic area is displayed where the roadway isassociated with a relatively low speed limit.

In one example, the action to be taken is to stop at an intersection andthe selected images include an icon indicating where the vehicle willstop at the intersection.

In one example, the action to be taken is waiting and the method furthercomprises displaying an icon indicating that the vehicle is waiting.

In one example, the processor is further programmed to determine thegeographic area to be displayed based on the action to be taken, whereif the action to be taken is a turn, the geographic area includes alarger view in the direction opposite to the turn.

In one example, the identified object is a traffic signal and theselected images include an icon indicating a traffic signal, and whereinthe processor is further programmed to display the icon on the displayproximate to the location of the traffic signal.

In one example, the identified object is a traffic signal and theselected images include an icon indicating a traffic signal, and theselected images include an icon indicating a state of the trafficsignal. The processor is also further programmed to determine the stateof the traffic signal; determine the action to be taken based on thestate of the traffic signal.

In one example, the action to be taken is changing to a different lane,and the selected images include an icon indicating a turn signal.

Another aspect of the invention provides a method for selecting imagesfor display on an display apparatus of a vehicle. The method includesreceiving destination information from a user input device; identifyinga route to the destination; receiving location information from ageographic position component; accessing stored map informationincluding roadways, traffic signals, and intersections; determining,from the location information and the stored map information, thecurrent geographic location of the vehicle; identifying an object of aroadway and an object type based on object information received from anobject detection apparatus; determining an action to be taken includingcontrolling at least one of a plurality of control apparatuses includinga braking apparatus, an acceleration apparatus, and a steeringapparatus, wherein the action to be taken is determined based on theidentified object, the current geographic location of the vehicle, andthe route; and selecting images to be displayed on the display apparatusbased on the action to be taken and the identified object.

In one example, the method further includes displaying the selectedimages on the display apparatus.

In one example, the method further includes determining a geographicarea to be displayed such that a larger geographic area is displayedwhere the vehicle is moving faster, and a smaller geographic area isdisplayed where the vehicle is moving slower.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional diagram of a system in accordance with an aspectof the invention.

FIG. 2 is an exemplary design of the interior of an autonomous vehiclein accordance with an aspect of the invention.

FIG. 3 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 4 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 5 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 6 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 7 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 8 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 9 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 10 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 11 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 12 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 13 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 14 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 15 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 16 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 17 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 18 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 19 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 20 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 21 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 22 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 23 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 24 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 25 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 26 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 27 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 28 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 29 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 30 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 31 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 32 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 33 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 34 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 35 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 36 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 37 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 38 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 39 is an exemplary screen shot in accordance with an aspect of theinvention.

FIG. 40 is an exemplary screen shot in accordance with an aspect of theinvention.

DETAILED DESCRIPTION

Aspects, features and advantages of the invention will be appreciatedwhen considered with reference to the following description of exemplaryembodiments and accompanying figures. The same reference numbers indifferent drawings may identify the same or similar elements.Furthermore, the following description is not limiting; the scope of theinvention is defined by the appended claims and equivalents.

As shown in FIG. 1, an autonomous driving system 100 in accordance withone aspect of the invention includes a vehicle 101 with variouscomponents. The vehicle may have one or more computers, such as computer110 containing a processor 120, memory 130 and other componentstypically present in general purpose computers.

The memory 130 stores information accessible by processor 120, includinginstructions 132 and data 134 that may be executed or otherwise used bythe processor 120. The memory 130 may be of any type capable of storinginformation accessible by the processor, including a computer-readablemedium, or other medium that stores data that may be read with the aidof an electronic device, such as a hard-drive, memory card, ROM, RAM,DVD or other optical disks, as well as other write-capable and read-onlymemories. Systems and methods may include different combinations of theforegoing, whereby different portions of the instructions and data arestored on different types of media.

The instructions 132 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computer codeon the computer-readable medium. In that regard, the terms“instructions” and “programs” may be used interchangeably herein. Theinstructions may be stored in object code format for direct processingby the processor, or in any other computer language including scripts orcollections of independent source code modules that are interpreted ondemand or compiled in advance. Functions, methods and routines of theinstructions are explained in more detail below.

The data 134 may be retrieved, stored or modified by processor 120 inaccordance with the instructions 132. For instance, although the systemand method is not limited by any particular data structure, the data maybe stored in computer registers, in a relational database as a tablehaving a plurality of different fields and records, XML documents orflat files. The data may also be formatted in any computer-readableformat. By further way of example only, image data may be stored asbitmaps comprised of grids of pixels that are stored in accordance withformats that are compressed or uncompressed, lossless (e.g., BMP) orlossy (e.g., JPEG), and bitmap or vector-based (e.g., SVG), as well ascomputer instructions for drawing graphics. The data may comprise anyinformation sufficient to identify the relevant information, such asnumbers, descriptive text, proprietary codes, references to data storedin other areas of the same memory or different memories (including othernetwork locations) or information that is used by a function tocalculate the relevant data.

The processor 120 may be any conventional processor, such as processorsfrom Intel Corporation or Advanced Micro Devices. Alternatively, theprocessor may be a dedicated device such as an ASIC. Although FIG. 1functionally illustrates the processor, memory, and other elements ofcomputer 110 as being within the same block, it will be understood bythose of ordinary skill in the art that the processor and memory mayactually comprise multiple processors and memories that may or may notbe stored within the same physical housing. For example, memory may be ahard drive or other storage media located in a server farm of a datacenter. Accordingly, references to a processor or computer will beunderstood to include references to a collection of processors orcomputers or memories that may or may not operate in parallel.

Computer 110 may include all of the components normally used inconnection with a computer such as a central processing unit (CPU),memory (e.g., RAM and internal hard drives) storing data 134 andinstructions such as a web browser, an electronic display 142 (e.g., amonitor having a screen, a small LCD touch-screen or any otherelectrical device that is operable to display information), and userinput (e.g., a mouse, keyboard, touch-screen and/or microphone).

Computer 110 may also include a geographic position component 144 todetermine the geographic location of the device. For example, computer110 may include a GPS receiver to determine the device's latitude,longitude and/or altitude position. Other location systems such aslaser-based localization systems, inertial-aided GPS, or camera-basedlocalization may also be used.

Computer 110 may also include other features, such as an accelerometer,gyroscope or other acceleration device 146 to determine the direction inwhich the device is oriented. By way of example only, the accelerationdevice may determine its pitch, yaw or roll (or changes thereto)relative to the direction of gravity or a plane perpendicular thereto.In that regard, it will be understood that a computer's provision oflocation and orientation data as set forth herein may be providedautomatically to the user, other computers of the network, or both.

Computer 110 may also include an object detection component 148 todetect and identify the location and movement (e.g. relative speed) ofobjects such as other vehicles, obstacles in the roadway, trafficsignals, signs, etc. The detection system may include lasers, sonar,radar, cameras or any other such detection methods. For example, theobject detector may include an imaging device to identify the state of aparticular traffic signal as yellow or another color. In use, computer110 may use this information to instruct the braking system of thevehicle to apply the brakes.

Data 134 may include various types of information used by computer 110.Detailed map information 136 may include maps identifying lane lines,intersections, speed limits, traffic signals, buildings, signs, or othersuch information. For example, computer 110 may access detailed mapinformation 136 in order to determine where the lane lines should belocated on a particular highway and adjust the speed or direction ofvehicle 101 accordingly. Computer 110 may also access display images138, such as roadways, intersections, and other objects in order toprovide a passenger of vehicle 101 with an understanding of what actionsvehicle 101 will take in the immediate future.

In one example, computer 110 may be an autonomous driving computingsystem capable of communicating with a vehicle's internal computer suchas computer 160. Computer 160 may be configured similarly to computer110, for example, including a processor 170, memory 172, instructions174, and data 176. Computer 110 may send and receive information fromthe various systems of vehicle 101, for example the breaking 180,acceleration 182, signaling 184, and navigation 186 systems in order tocontrol the movement, speed, etc. of vehicle 101. It will be understoodthat although various systems and computers 110 and 160 are shown withinvehicle 101, these elements may be external to vehicle 101 or physicallyseparated by large distances.

FIG. 2 depicts an exemplary design of the interior of an autonomousvehicle. Vehicle 101 may display information to a passenger in a numberof ways. For example, vehicle 101 may be equipped with a display 242 forthe autonomous driving system and an indicator 255 which identifieswhether the autonomous driving system has been engaged. Vehicle 101 mayalso identify the current speed of travel 245 by displaying theinformation in a location visible to the passenger or identifying thespeed audibly. Further, as shown in the figure, vehicle 101 may alsoinclude a second display 286 for a navigation or entertainment system.

Vehicle 101 may include one or more user input devices, such as device240, for inputting information into the autonomous driving computer 110.For example, a user may input a destination, (e.g. 123 Oak Street), intothe navigation system. The navigation system may generate a routebetween the present location of the vehicle and the destination. If theautonomous driving system is engaged, computer 110 may request orautomatically receive the route information from the navigation system.Once a route has been determined, the autonomous driving system maydrive the vehicle to the destination.

FIG. 3 is an exemplary screen shot of the display 242 of computer 110.The display may be used to identify to the passenger of the vehicle thecurrent speed, location, orientation, etc. of vehicle 101 with respectto a relevant stretch of roadway 305. For example, vehicle 101 isdepicted as an icon or box 310 on the screen. It will be understood thatany relevant object may be used to identify the vehicle to the user. Theroute along which the vehicle is traveling may be identified by a lineextending from the vehicle. In the example of FIG. 3, the vehicle ismoving forward, so route line 315 extends from the front of the vehicle.The display may also identify the speed indicator 345 of the device aswell as the approximate path of the vehicle during the next few seconds,shown as path 355.

As shown in FIG. 4, the display may also identify other relevantobjects. For example, where computer 110 has identified another vehicleon the roadway, the computer may display the vehicle as another icon,such as box 410, on the display in the approximate location of theroadway. The shape and size of the icon displayed may be selected bycomputer 110 based on the shape and size of actual vehicle, i.e. acompact vehicle may appear smaller than a tractor-trailer.

As vehicle 101 moves along the roadway, the location of objects detectedby the vehicle, and the features of the roadway may change. Thesechanges may be displayed in order to allow the user to understand thatvehicle 101 is continuously monitoring the state of the vehicles,roadway and other objects. For example, as shown in FIG. 4, vehicle box310 is driving along the roadway approaching a second vehicle identifiedas box 410. Some time later, vehicle 101 moves closer to the secondvehicle, and as shown in FIG. 5, the distance between vehicle box 310and box 410 has decreased. In addition, bar 525 may indicate that bothvehicles are approaching an intersection.

As vehicle box 310 approaches an intersection, computer 110 may providethe user with additional information. For example, display may nowinclude a representation of a cross-roadway 605 as well as additionalvehicles, depicted as boxes 620 and 630. In addition, as computeridentifies vehicles on roadway 305, these new vehicles may also bedisplayed as boxes, such as box 610.

Computer 110 may also decrease the speed of vehicle 101 as it approachesthe intersection, thus, the current speed of vehicle 101 may becontinuously updated. For example, as shown in FIGS. 4 and 5, vehicle101 is moving at 34 miles per hour, as indicated by speed indicator 345.As shown in FIG. 6, vehicle 101 is much closer to the intersection, thuscomputer 110 may slow vehicle 101's speed, thus speed indicator 345 nowdisplays a speed of 27 miles per hour.

Computer 110 may also pan and zoom the display based on the type ofroadway, speed limit of the roadway, and/or speed of the vehicle. Forexample, as shown in FIG. 7, vehicle 101, displayed as box 310, ismoving along roadway 705 at 65 miles per hour. Thus, computer 110displays a relatively large stretch of roadway 705. Returning to FIG. 4,vehicle 101 is moving at only 34 miles per hour, thus, computer 110 maydisplay less of roadway 305. Similarly, the display may pan and zoombased on the speed limit of the roadway. In another example, if vehicle101 is moving along a highway with cross traffic, display may zoom outmore than if vehicle 101 is stopped at an “four-way” stop intersection,as the passenger may only need to see the intersection some relativelyshort distance around the intersection. In another example, if vehicle101 is making an unprotected left-hand turn, the display may zoom out toa relatively large distance so that the passenger may see informationfrom all directions.

As vehicle 101 approaches other vehicles, computer 110 may maintain asafe following distance and display this information to the passenger.As show in FIG. 8, box 310 is moving along roadway 805 at 33 miles perhour. As box 310 approaches box 810, path 855 may shorten in order toindicate this information to the passenger. Box 810 may include a tail865. The path 855 and tail 865 may be displayed as complementary shapes,for example as two arrow portions which may be fit together. In FIG. 9,box 310 has moved closer to box 810, and path 855 and tail 865 aredisplayed together. The closing of these two shapes may indicate thatbox 310 has reached a safe following distance behind box 810. The safefollowing distance may change based on the speed of vehicle 101, avehicle in front of 101, and/or the speed limit of the roadway.

If the vehicle represented by box 810 slows down, computer 110 maydetermine that vehicle 101 is no longer following at a safe distance.Computer 110 may determine that vehicle 101 must slow down in order tocorrect the distance and display this information to the passenger. Forexample, FIG. 10, the shapes of path 855 and tail 865 have changeddirection, which may indicate that vehicle 101 is not located a safedistance behind the next vehicle or box 810. Thus, computer 110 mayreduce the speed of vehicle 101 and indicate this change to the user.For example, in FIG. 9, vehicle 101 is moving at 28 miles per hour,while in FIG. 10, vehicle 101 is moving at 22 miles per hour. Inaddition, as shown in FIG. 9, computer 110 may display an icon, such asdouble arrows, with speed indicator 345. In the example, the icon isdouble arrows directed upwards. In FIG. 10, as computer 101 has reducedthe speed of vehicle 101, the icon displayed may be changed, and, forexample, the arrows may be directed downwards.

Computer 110 may determine the safe following distance based on thespeed of vehicle 101, and may display the distance and zoom levelaccordingly. For example, as shown in FIGS. 10 and 11, where the vehicle110, shown as box 310, is moving at 22 miles per hour, the safefollowing distance, 855 and 865, is much less than the safe followingdistance, 1155 and 1165, where the vehicle is moving at 65 miles perhour. Thus, comparing FIGS. 10 and 11, where the vehicle is moving at aslower speed, computer 110 may display a smaller area and length ofroadway, whereas, if the vehicle is moving faster, a greater area andlength of roadway may be displayed. In another example, the safefollowing distance may be based on the speed limit of the roadway.

Computer 110 may also display information about intersections to thepassenger. For example, as shown in FIG. 12, box 310 is approaching anintersection with a stop sign. For example, as discussed above, bar 1225may indicate an intersection, and computer 110 may display an icon, suchas stop sign 1285 along roadway 1205 to indicate that vehicle 101 isapproaching a stop. Path 1255 includes a section 1275 past bar 1225which may indicate that when vehicle 101 reaches the location of bar1225, computer 110 may stop vehicle 101 for some period of time. Forexample, section 1275 may be displayed differently from the remainder ofpath 1255, such as stationary or moving stripes. Computer 110 may alsoidentify and display other vehicles on the roadway, for example box1210.

As shown in FIG. 13, when vehicle 101 reaches the location of bar 1225,vehicle 101 may come to a stop to wait until it is safe to cross theintersection, as indicated by the striped path 1275. In the example, box1210 is moving along roadway 1305 and is near the intersection. In FIG.14, both boxes 310 and 1210 are displayed at the intersection. In oneexample, computer 110 may display an icon, such as a ticking clock,indicating that box 1210 has the right of way, and computer 110 willwait for box 1210 to take some action. Computer 110 may also wait for aspecified period of time, and if box 1210 has not yet moved, thecomputer may determine that box 1210 is waiting for vehicle 110 and movevehicle 110 as required by the route. As shown in FIG. 15, box 1210 hasmoved through the intersection. As the intersection is now clear,vehicle 101 moves through the intersection and box 310 is shown movingthrough the intersection as well. Path 1255 is displayed as a solid bar,which may indicate that computer 110 is no longer stopped or waiting.

In the example of FIG. 16, box 310 is waiting at an intersection to makea right turn. Speed 345 may display information indicating that thecomputer is waiting, such as the text “waiting.” As described above,computer 110 has panned the display to the right vehicles, such as box1610 moving along roadway 1605 and to the right of box 310 are moreimportant to the passenger than objects moving along roadway 1607 to theright of box 310. Path 1675 may also indicate that vehicle 101 iswaiting for the intersection to be safe for a right turn. As shown inFIG. 17, as box 1610 approaches box 310, computer 110 may display a tail1765 which complements the shape of path 1675. Once box 1610 has passedbox 310, as shown in FIG. 18, path 1875 may be displayed as solid toindicate that vehicle 101 is ready to make the right turn. As describedabove, when vehicle 101 makes the turn, path 1875 and tail 1765 may cometogether, and computer 110 may maintain a safe following distance behindbox 1610.

FIG. 19 depicts a more complicated left hand turn across several lanesof traffic. Similar to the example of FIGS. 16-18, computer 110 may waitto turn until the intersection is safe. Computer 110 may wait until bothboxes 1910 and 1920 have cleared path 1975. As shown in FIG. 20, box1910 has moved past path 1975, however box 1920 has not, thus computer110 may continue to display the “waiting” text and the path 1975 asstriped. As box 1920 approaches path 1975, computer may determine thatit is safe to begin turning. Thus, as shown in FIG. 21, path 1975 nowpath 2155 is displayed as solid and box 1920 includes a tail 2165. Speed345 may no longer display the waiting text and indicate that vehicle 101is moving, for example at a speed of 10 miles per hour. As vehicle 101makes the left turn, as shown in FIG. 22, display may move to the leftto indicate the change in direction. Box 310 may approach tail 2165 ofbox 1920, and as shown in FIGS. 23 and 24, the complementary shapes mayclose together as vehicle 101 reaches a safe following distance, and thedisplay may continue to shift the location of the roadway.

Computer 110 may also identify, recognize, and display the state oftraffic signals. As shown in FIG. 25, vehicle 101 is approaching anintersection and computer 110 is planning to make a left turn. Aninverted arrow icon 2557, may be displayed to indicate that computer 110is or will cause vehicle 101 to slow down based on the intersectionahead. Computer 110 may determine, based on the detailed map that theintersection includes a number of traffic signals. In FIG. 25, computermay have identified a traffic signal but may not have determined thecurrent state of the traffic signal. Computer 110 may indicate thisinformation to the passenger by displaying an icon such as icon 2585. Itwill be understood that the polygon and question mark icon are onlyexemplary and any such indicator may be used. As shown in FIG. 26,computer 110 has determined the state of the traffic signal, and mayindicate the state at icon 2685. For example, if computer 110 hasdetermined that the traffic signal is red or yellow, path 2555 mayinclude a striped section 2675 past bar 2525. Striped section 2675 whichindicates that computer 110 may stop vehicle 310 at bar 2525 and wait atthe intersection until it is safe to make the left turn.

Computer 110 may also display a headroom zone in front of boxesindicating other vehicles in order to inform the passenger that thecomputer will not move into this zone. As shown in FIG. 27, vehicle 101is approaching an intersection, as indicated by bar 2725. The shape ofpath 2755 indicates that computer 110 will make vehicle 101 perform aleft turn. Striped section 2775 of path 2755 indicates that once vehicle101 reaches the intersection, computer 110 will wait until it is safe tomake the left turn. In the example, computer 110 has identified avehicle, displayed as box 2710 which will pass through path 2755.Computer 110 may display a headroom zone 2735 which computer 110 willavoid in order to move through the intersection or make the left handturn safely.

As shown in FIG. 28, box 2710 has moved closer to box 310, and stripedsection 2775 is no longer displayed as computer 110 has determined thatit is now safe to make the left turn. Thus computer 110 may begin toincrease the speed of vehicle 101. In addition, as box 310 is no longerlikely to move into the headroom zone of box 2710, computer 110 may nolonger display headroom zone 2735. In the example, a new vehicle,displayed as box 2810, is approaching path 2755. Computer 110 may alsodisplay a headroom zone 2835 in order to indicate to the passenger thatthe computer will not move vehicle 101 into this zone. Again, ifcomputer 110 determines that it is safe to make the turn despite thecontinued approach of the headroom zone 2835 of box 2810, shown in FIG.29, computer 110 will direct vehicle 101 to make the left turn.

Computer 110 may also use the display to indicate to the passenger thatthe vehicle 101 will be changing lanes. As shown in FIG. 30, vehicle 101is moving along route 3015 behind a second vehicle, displayed as box3010. The positions of path 3055 and tail 3065 may indicate that vehicle101 is maintaining a safe following distance along route 3015 behind thesecond vehicle. If computer 110 determines a change of lanes isappropriate as shown in FIG. 31, the computer may display a turn signalicon 3195 which indicates that the vehicle is going to move to the laneto the right of box 310, for example if route 3015 requires the vehicleto take an exit or make a turn. In addition, computer 110 may display aturning zone 3175, similar to a path with stripes, to indicate thatcomputer 110 is waiting until it is safe to move into the zone. In theexample, computer 110 is waiting because the headroom zone 3135 of athird vehicle, displayed as box 3020, is within the turning zone 3175.If computer has determined that vehicle 101 will follow behind the thirdvehicle, computer 110 may also display a tail 3165 behind box 3020. Asshown in FIG. 32, as vehicle 101 begins to slow down and move towardsthe tail 3165, computer 110 may begin to fade the display of headroomzone 3135 until, for example, the headroom zone is no longer useful orneeded as shown in FIG. 33.

Similarly, computer may also indicate to the passenger that the vehicleis ready to turn. Once vehicle 101 moves behind the third vehicle, andbox 3020 is no longer within the turning zone 3175, computer 110 may nolonger display the turning zone 3175, as shown in FIG. 34. In addition,route 3015 may be repositioned to show that vehicle 101 is going to movealong the roadway in the lane to the right of box 310. As shown in FIG.35, computer 110 may begin to move vehicle 101 into the lane to theright, and on the display move the end of complementary ends of path3055 tail 3165 of box 3020 towards one another as described above. Asvehicle 101 has not yet completely changed lanes, computer 110 maycontinue to display turn signal icon 3195. Then, as shown in FIG. 36,when vehicle 101 moves completely into the lane, computer 110 may fadeor remove turn signal icon 3195.

Computer 110 may display turning information in various additional ways.For example, as shown in FIG. 37, computer 110 may determine thatvehicle must move one lane to the left in order to avoid the vehiclesdisplayed as 3710, 3720, and 3730. Computer 110 may display a small icon3795 close to box 310 indicating that computer 110 is planning to movevehicle 101 one lane to the left. In addition, as shown in FIG. 38,computer 110 may also move path 3755 and route 3715 into the left laneto indicate the future path of vehicle 101.

Computer 110 may also display information such as an estimated time to aparticular location. For example, as shown in FIG. 39, Vehicle 101 ismoving along roadway 3905, and route 3915 requires that the vehicle turnat the intersection with roadway 3907. Computer 110 may display theestimated time for vehicle 101 to reach a relevant location, forexample, the end of path 3955, the intersection of roadways 3905 and3907, or the next turn of route 3915.

Computer 110 may use the display to convey other relevant information tothe passenger. For example, as shown in FIG. 40, vehicle 101 is movingalong an overpass. Computer 110 may display box 310 moving along aroadway 4005 over roadway 4007. Computer 110 may display the vehicles aswell as the roadway underneath vehicle 101 in such a way as to indicatethat these objects are below vehicle 101. For example, roadway 4007 andthe vehicles moving along the roadway, for example boxes 4010 and 4020,may be displayed as faded or darker to indicate to the passenger thatthese objects are below vehicle 101 or are simply less relevant for someother reason.

As these and other variations and combinations of the features discussedabove can be utilized without departing from the invention as defined bythe claims, the foregoing description of exemplary embodiments should betaken by way of illustration rather than by way of limitation of theinvention as defined by the claims. It will also be understood that theprovision of examples of the invention (as well as clauses phrased as“such as,” “e.g.”, “including” and the like) should not be interpretedas limiting the invention to the specific examples; rather, the examplesare intended to illustrate only some of many possible aspects.

It will be further understood that the sample values, icons, types andconfigurations of data described and shown in the figures are for thepurposes of illustration only. In that regard, systems and methods inaccordance with aspects of the invention may include different physicalattributes, data values, data types and configurations, and may beprovided and received at different times and by different entities(e.g., some values may be pre-suggested or provided from differentsources).

The invention claimed is:
 1. A method of providing for display an image representing a driving environment of a vehicle to an occupant of the vehicle, the method comprising: determining, by one or more processors, a geographic location of the vehicle; receiving, by the one or more processors, from a sensor of the vehicle information identifying an object in the vehicle's external environment; identifying, by the one or more processors, one or more features expected to be in the vehicle's external environment based on the determined location; generating, by the one or more processors, the image for display including a representation of the vehicle, a representation of the object, and a background scene corresponding to the identified features, wherein a coloring of the representation of the object indicates a degree of relevance of the object to the vehicle; providing, by the one or more processors, the image for display to the occupant of the vehicle.
 2. The method of claim 1, wherein the representation of the object indicates that the object is below the vehicle.
 3. The method of claim 2, wherein the information identifying the object indicates that the object is located below an overpass and the method further comprises determining that the vehicle is located on the overpass based on the determined location.
 4. The method of claim 1, wherein the coloring of the representation of the object as compared to a coloring of the representation of the vehicle indicates that the object is below the vehicle.
 5. The method of claim 1, wherein a shading of the representation of the object as compared to a shading of the representation of the vehicle indicates that the object is below the vehicle.
 6. The method of claim 5, wherein the shading of the representation is lighter than the shading of the representation of the vehicle.
 7. The method of claim 1, wherein the information identifying the object indicates that an object type of the object is a vehicle and the representation of the object indicates that an object type of the object representation is a vehicle.
 8. The method of claim 1, wherein the representation of the object indicates that the object is on a first roadway of the background scene and the representation of the vehicle indicates that the vehicle is on a second roadway of the background scene different from the first roadway.
 9. A system for providing for display an image representing a driving environment of a vehicle to an occupant of the vehicle, the system comprising: memory storing map information identifying features expected to be detected at various locations; one or more processors configured to: determine a geographic location of the vehicle; receiving from a sensor of the vehicle information identifying an object in the vehicle's external environment; identify one or more features expected to be in the vehicle's external environment based on the determined location; generate the image for display including a representation of the vehicle, a representation of the object, and a background scene corresponding to the identified features, wherein a coloring of the representation of the object indicates a degree of relevance of the object to the vehicle; provide the image for display to the occupant of the vehicle.
 10. The system of claim 9, further comprising: the vehicle; a display configured to display the image.
 11. The system of claim 10, wherein the representation of the object indicates that the object is below the vehicle.
 12. The system of claim 11, wherein the information identifying the object indicates that the object is located below an overpass and the method further comprises determining that the vehicle is located on the overpass based on the determined location.
 13. The system of claim 10, wherein the coloring of the representation of the object as compared to a coloring of the representation of the vehicle indicates that the object is below the vehicle.
 14. The system of claim 10, wherein a shading of the representation of the object as compared to a shading of the representation of the vehicle indicates that the object is below the vehicle.
 15. The system of claim 14, wherein the shading of the representation is lighter than the shading of the representation of the vehicle.
 16. The system of claim 10, wherein the information identifying the object indicates that an object type of the object is a vehicle and the representation of the object indicates that an object type of the object representation is a vehicle.
 17. The system of claim 10, wherein the representation of the object indicates that the object is on a first roadway of the background scene and the representation of the vehicle indicates that the vehicle is on a second roadway of the background scene different from the first roadway.
 18. A non-transitory computer-readable medium on which instructions are stored, the instructions, when executed by one or more processors, cause the one or more processors to perform a method of providing for display an image representing a driving environment of a vehicle to an occupant of the vehicle, the method comprising: determining a geographic location of the vehicle; receiving from a sensor of the vehicle information identifying an object in the vehicle's external environment; identifying one or more features expected to be in the vehicle's external environment based on the determined location; generating the image for display including a representation of the vehicle, a representation of the object, and a background scene corresponding to the identified features, wherein a coloring of the representation of the object indicates a degree of relevance of the object to the vehicle; providing the image for display to the occupant of the vehicle.
 19. The medium of claim 18, wherein the information identifying the object indicates that the object is located below an overpass and the method further comprises determining that the vehicle is located on the overpass based on the determined location.
 20. The medium of claim 18, wherein a shading of the representation of the object as compared to a shading of the representation of the vehicle indicates that the object is below the vehicle. 